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ABSTRACT 

It has been shown that Guttman weighting of test 
options results in marked increases in the internal consistency of a 
test. However, the effect of this type of weighting on the structure 
of the test is not known. Hence, the purpose of this study is to 
compare the factor structure of Guttman-weighted and 
rights- only— weighted tests and to relate the change in structure to 
the change in internal consistency. Test items were intercorrelated 
and the resultant matrices factored, first weighting with Guttman 
weights then with rights-only ones. More variance was accounted for 
by the first several factors of Guttman-weighted tests than by these 
same factors in rights-only weighted tests. The tests for which there 
was most increase in variance were those for which there was greatest 
increase in internal consistency. Finally, the factor structure and 
conventionally-weighted tests were quite different. The implication 
of these structural changes on test validity is discussed. 
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Comparison of the Factor Structure of 
Guttman-Weighted ve. Rights -Only-Weighted Tests 

Abstract 

It has been shown that Guttman weighting of test options results 
in marked increases in the internal consistency of a test* However, 
the effect of this type of weighting on the structure of the test 
is not known. Hence, the purpose of this study is to compare the 
factor structure of Guttman-weighted and rights-only-weighted tests 
and to relate the change in structure to the change in internal 
consistency. 

Test items were in ter correlated and the resultant matrices 
factored, first weighting with Guttman weights then with rights- 
only ones. More variance! was accounted for by the first several 
factors of Guttman-weighted tests than by these same factors in rights- 
only-weighted tests. The tests' for which there was most increase in 
variance were those for which there was greatest increase in internal 
consistency. Finally, the factor structure and conventionally-weighted 
tests was quite different. The Implication of these structural changes 
on test validity sis discussed. 
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Comparison of the Factor Structure of 

* 

Guttman-Weighted vs. Rights -Only-Weighted. Teats 

This research is an outgrowth of differential option weighting studies 

reported by Davis (1959), Davis and Fifer (1959), Sabers and White (1969), 

and Hendrickson (1971), with the latter . study being most directly related. • 

Hendrickson (1971) used an empirical weighting technique described by 

Guttman (1941) to assign values to the options in each subtlest of the : 

Scholastic Aptitude Test (SAT) . Guttman^ technique is designed to maxi- 

jaize the internal-consistency reliability ..(coefficient «)»- 

It was found, as expected, that the internal-consistency reliability 

of each subtest increased substantially as a result of Guttman weighting. 

« 

However, the intercorrelation of all subtests, except the two verbal 
subtests,' decreased. These results invited the interesting speculation 
that the underlying structure of a test changed when scored with Guttman 
weights. In other words, the Guttman-weighted and conventionally-weighted 
tests were measuring different things. The following line of reasoning 
led to this hypothesis. 

If the internal-consistency coefficient is greater for a Guttman- 
weighted test than for a conventionally-weighted test, and if the loading 
pattern of items on factors is similar in the two cases, then the former 
test is measuring the same function as the latter test, but measuring it more 

efficiently. Therefore, the. former test should^ correlate bhtte'r -with' another 
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test or external criterion variable than the latter. If| however, 
the loading pattern of items on factors is different, then the two 
tests are measuring different functions. In this case it is likely 
that a Guttman-weighted test will correlate less well with another 
test or outside criterion than a conventionally-weighted test. The 
fact that the values of the intercorrelation coefficients of the 
Guttman-weighted sub tests of the SAT decreased is, therefore, strong 
evidence that the structure of the subtests was altered. 

The purpose of this study is to compare the factor structure of 
Guttman-weighted and rights -only -weigh ted SAT. A comparison of the 
factor structure will point out the effect of this type of weighting 
on the underlying structure of the test. Such information should 
make "blind" empirical weighting somewhat less blind. 

PROCEDURE 

The sample used in this study consisted of 200 boys and 200 girls, 
randomly selected from the examinees retained by Educational Testing 
Service for item-analysis purposes. , The test used was form QSA43 of 
the SAT, administered at the College Entrance Examination Board’s 
regular testing in November, 1968. All four subtests in the scored 
portion of the SAT were used. These subtests are numbered 1, 2, 4, 
and 5. Sub test 3 is not part of the scored pprtion of the SAT; it 
is used only for equating and pretesting purposes. Hence, it was 

not included in this study. 

$ 
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The verbal section comprises subtests 1 (40 items) and 2 (50 items); 
the mathematical comprises subtests 4 (35 items) and 5 (25 items). 

Guttman weights were calculated for all options in the SAT 
in the way described by Hendrickson (1971). These weights were 
determined separately for each subtest, the weight of a particular 
option being the total score on the other items of the subtest for 
all examinees who marked the option in question. The weight for 
"omit" was computed in the same manner as the weights of the other 
five options in an item. Crossvalidation of the weights, a pro- 
cedure which usually should be carried out when dealing with 

the- empirically-calculated Guttman weights, was deemed inappropriate 
3 

inr. this study. 

The comparison of the factor structure of Guttman and conven- 
tionally-weights tests involved two investigation- -the first 
design to compare the difference in common variance and in variance 
per factor and the second designed to compare the loading pattern 
of items on factors. In the first investigation the basic unit 
of analysis was the inter-item, product-moment correlation matrix. 

Two inter-item correlation matrices were calculated for each subtest 
of the SAT! one consisting of the . • cor re 1 a t i on coefficients of items 
with Guttman weights (a six-category distribution for each item). 



3 The intent of this study is merely to show how weighting might effect 
the structure of a test. Maximum structural changes will result when 
item weights are calculated on the sample of individuals whose tests 
are then factored. The extent to which these structural changes 
exist when weights are calculated on another random sample from the 
population depends on the extent to which the two samples are similar. 
The blurring effects of sampling error is undesirable in this instance* 
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and the other consisting of correlation coefficients of items 
weighted dichotomously (1 or 0 ). These correlation matrices 
were factored using the Minres procedure (Harman and Jones, 1966 ; 
Harman, 1968 , pp. 187 - 211 ). Eleven factors were extracted. This 
number was chosen, somewhat arbitrarily, because in most cases 
all factors with eigenvalues greater than one were among the 
eleven. Further, eleven seemed to be a sufficiently large number 
to allow trends to appear. 
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Th@ coefficients .used in the first analysis, product -moment correlation 
coefficients, are calculated from the actual item weights from which the 
total test score is formed. Hence, these coefficients are the appropriate 
ones to use when comparing common variance and factor variance of un- 
rotated factors . However, a difficulty factor often emerges in analyses 
using product -moment correlation coefficients of dichotomous ly scored 
variables. Thus in the second analysis, designed to compare the loading 
pattern of items on factors , matrices of tetrachoric correlation coefficients 
were used for the dichotomously scored variables in order to lessen the 

effects of a difficulty factor. Matrices of product -moment correlation ' 

' * * 

coefficients were again used for Guttman-weighted tests. Factors were 
obtained by the Minres procedure, followed by a varimax (Kaiser, 1958) 
rotation. . 

* • 

t 

RESULTS AND DISCUSSION 

The results of the factor analysis in the first investigation are 
displayed in Table 1. The first row under each subtest contains the 









Insert Table 1 about here 

variance accounted for by the first eleven factors extracted from the 
inter-item, product-moment correlation matrix of* the Guttman-weighted 
subtest. The second row contains the same thing for the rights-only- 

weighted subtest . At the end of each row is the total variance accounted 
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for by the eleven factors. In all cases the amount of total 
variance was substantially larger for a Guttman-weighted sub- 
test than for the corresponding rights-only-weighted sub test. 

Also, a larger amount of variance was accounted for per factor 
for the first several factors in a Guttman-weighted sub test. 

t 

The intemal-rconsistency reliability coefficients for 
Guttman- and rights-only-weighted. subtests are listed in the 
last column of Table 1, along with the percent by which a rights- 
only-weighted test would have to be lengthened (as calculated by 
the Spearman-Brown prophecy formula) to account for the increase 
in internal consistency that came about as a result of Guttman 
weighting. It is important to note that the tests for which 
there is greatest increase in variance are the ones for which 
there is greatest increase in internal-consistency reliability 
(as measured by effective increase in test length). This finding 
is thoroughly predictable because an increase in internal con- 
sistency without adding more items implies that the mean intraclass 
coefficient among thet existing items has gone up (Stanley, 1957 
and 1971). Thus, items must share more variance in common. 
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Comparison of the varimax factor matrices obtained by rotating the 
first eleven Minres factors of the tctrachoric correlation matrix shows 

that the structure of Guttman- and • rights -only-weighted subtests are 

* ' 

considerably different. 4 In only one case (Guttman-weighted subtest 1) 
was there a clear-cut difficulty factor. In this case the difficulty, 
factor was the one which accounted for the next-to-the-most variance. 

In four of the eight factor matrices a speed factor emerged; however, 
some confounding was present in two cases. Three of the speed factors 
emerged in the Guttraan-weigihted subtests, in all cases as the factor 
which accounted for most variance. Thus Guttman weighting may be 
capitalizing on speed at the expense of intellectual variables. (In 
the verbal sub tests the moat difficult items are often not the last ones, 
making it possible to distinguish between difficulty and speed factors. 

In the mathematical sub tests, on the other hand, the last items are usually 
among the most difficult ones; thus, the naming of speed and difficulty 
factors must be more tentative in these two subtests.) Further attempts 
to name the factors were fruitless. Suffice it to say that the under- 
lying structures of both the Guttman- and rights-only-weighted-SAT-;^ 
are extremely complex. 



These eight factor matrices are displayed in a Johns Hopkins 

technical report which can he requested from the author. 
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The findings of this study may offer interesting implications 
for validity. It is likely that the structural changes imposed by 
differential weighting for internal consistency will probably make 
a test correlate I less welL. with an outside criterion, such as grade 
point average. Guttman (1941, pp. 296-297) himself predicted this 
outcome, and his prediction was bom out almost completely in 
Hendrickson's (1971) study. Hendrickson intercorrelated the four 
subtests of the SAT (6 combinations) and found that in five of the 
six cases the subtests correlated less well with each other when . 
they were weighted with Guttman weights than when they were weighted 
rightse-only. The one case in which the two subtests correlated more 
highly when weighted with Guttman weights involved the two verbal 
sub tests. It is noteworthy that this pair of sub tests are more 
alike in item type, item difficulty, and item content than the sub- 
tests of any other pair. A Guttman-weighted test will correlate 
better with an outside criterion only if the structural changes 
brought about by the weighting make the test structurally more 
like the outside criterion. This outcome is unlikely unless the 
tests are extremely similar to begin with, especially since Guttman 
weighting appears, in some cases, to capatalize on spurious factors, 
such as speed. Thus, it seems to be a good bet that, in most cases, 
even though the homogenity of a Guttman-weighted test will go up, 
the validity will go down. 

This conclusion directly contradicts the prediction expressed 
in the correction- for-attenuation formula, which says that the 
correlation coefficient between two tests increases If the reliability 
of either or both of the tests increases. Some twenty years ago 

: ... ..v^ivy ' 10 
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Loevinger and her colleagues (Loevinger, 1954; Loevinger, Gleser, 
and DuBois, 1953) noted and commented extensively upon such 
"attentuatictn paradoxes." Recently Gleser (1971) shed further 
light on the subject. The simple way to resolve the apparent 
"paradox" in this case is to remember that the correction- for- - 
attenuation formula is only appropriate if the test measures the 
same thing after the reliability is increased as it did before, 
i.e., if the loading pattern of items on factors has not changed, 
the loadings are merely higher. The correction-for-attenuation 
formula is completely inappropriate in cases when structural 
changes have occurred. 

Other. researchers r have <?used slightlyt different] versions of t the 
empiricaliweigh ting: scheme described by Guttman. The essence of 
this class of empirical weighting techniques is that the weight of 
an option is based on the scores of people marking that option on a 
criterion score distribution. For Guttman (1941) and Hendrickson 
(1971) the criterion was the score distribution of the test in 
question; their aim was to improve internal consistency. For Davis 
and Fifer (1959) the criterion was the score distribution of a parallel 
form of the test in question; their aim was to improve parallel-forms 
reliability. For Sabers and White (1969) the criterion was the score 
distribution on an outside achievement test; their aim was to improve 
validity. It should be emphasize that tests weighted by any of these 

methods will almost certainly be different structurally than a rights- 

\ ■ ' 

only-weighted test. 
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One sometimes hears the argument that it is desirable to devise 
ways of improving the reliability of a test* If this could be 
accomplished, the argument goes, more measuring ability could be 
packed into less test- taking time, thus leaving time for the measure 
ment of other attributes. This study has shown that the former line 
of reasoning is not always an appropriate one, especially if it is 
an internal- consistency coefficient which is increased and if the 
increasing is done by an empirical weighting procedure* Blind 
empirical weighting, indeed, might cause the intemalcconsistency 
to increase, but it also might result in structural changes which 
make the test measure something altogether different. 



u 



12 



11 



REFERENCES 



- av ,. v -a Estimation and use of scoring weights for each choice in 
D iultiplc-choice test items . Educational and Psychological Measure- 
ment, 1959, 12, 291-298. 

Davis F. B. & Fifer, G. The effect on test reliability and validity of 
scoring aptitude and achievement tests with weights every choxce. 
Educational and Psychological Measurement, -1959, i2, 159-1™ • 

Gleser L. J. The attenuation paradox and internal consistency. Presented 
at the American Educational Research Association meeting, February 

1971. 

Guttman, L. The quantification of a class of attributes: * theory and 

method of scale construction. In Paul Horst, The predicti on 
■personal ad.iustment . N. Y. : Social Science Research Council, 1941. 

Harman, H. H. Modern Factor Analysis . (2nd ed.) Chicago: The University 

of Chicago Press, 19&8. ' 1 ^ 

Harman, H. H. & Jones, W. H. Factor analysis by minimizing residuals 
(Minres). Psychoraetrika, 1$66, 31, 351-3&0. 

Hendrickson, 0. P. The effect of differential option ^weighting on multiple- 
choice objective tepts. Journal of Educational Measurement, 1971- 
(In press) 

Kaiser, H. • The varimax r Iterion for analytic rotation in factor analysis. 
Psvchometr ika , 1958, 23, 187-200. 

Loevinger, J. The attenuation paradox in test theory. Psychological 
Bulletin^ 1954, **93-504. 

Loevinger, J., Gleser, G. 0., & DuBois, P. H. Maximizing the discrindnating 
power of a multiple-score test. Psychometnka , 1953, 1 2, 309-317 • 

Sabers, D. L., & White, B. W. The effect of differential weighting of 

individual item responses on the predictive validity and reliability 
of an aptitude test. Journal of Educational Measurement, 19®9, £, 

93-96. 

Stanley, J. C. K-R20 as the stepped-up mean item intercorrelation. 

Pp. 78-93 in the l4th Yearbook of NCMS , 1957* 

Stanley, J, 0. Reli abili ty, Oh. 13 la »• Thorndike (Ed.) gfoWH9P a A 
Measurement (2nd ed.) Washingtons American Council on. Education, 

19714 ■ ' 



13 



O 

ERIC 



H* W 
ft Hi 

Hi 

ft ft 

ft o 

ft ft 
ft H* 
< 

H ft 
ft 

<S B* 

? S 

• P 
o> 
w 
ft 



O 

ERIC 





c n 




CO 




CO 




CO 






C 




c 




c 




c 






o* 




o* 




o* 




o* 






& 


O ft 


& 


O rr 


5ft 


O ft 


5d 


O ft 






b* 


C ft 


H* 


C ft 


H* 


C ft 


b* 


C ft 






OQ 


ft in 


JO 


rr co 


)Q 


ft ft 


ro 


ft in 






ft* 


rt rt 


ft* 


rr rr 


ft* 


ft ft 


ft* 


rr rr 






rr 


9 


ft 


g 


ft 


s 


rr 


9 






ft 


ft On 


in 


ft -ft* 


ft 


ft to 


ft 


ft 








ft 


I 


ft 


1 


ft 


i • 


ft 










0 




O 




0 












ft 




ft 




ft 












b* 




H* 




b* 












< 








< 








-tv 


Ul 




ON 


Ui 


00 


Ui 


v4 






• 


• 


• 


• 


• 


• 


• 


• 






M 


o 


vi 


ON 


VO 


o 


v| 


4V 






04 


ON 


O 


b* 


VO 


-IV 


O 


-tv 








b* 


H 


ro 


b* 


ro 


H 


b* 






• 


• 


• 


• 


• 


• 


• 


• 






00 


00 


04 


H 


b* 


04 


04 


Ul 


|n4 




Ul 


v4 


On 


VO 


O 


b* 


Ul 


VO 






• 






M 


H* 


b* 


b* 








• 


• 


• 


• 


• 


• 


• 


* 






v4 


00 


vj 


ON 


O 


-tv 


O 


ro 


04 




ro 


b* 


ON 


Ul 


ro 


b* 


04 


v4 
















b* 




b* 






• 


• 


• 


• 


• 


• 


• 


• 






ON 


ON 


v| 


VO 


VO 


O 


v| 


b* 


-IV 




00 


VO 


H 


Ul 


00 


-tv 


ON 


04 






• 


• 


• 


• 


• 


• 


• 


• 






On 


On 


ON 


ON 


00 


VO 


ON 


ON 


On 


• 


v| 


ON 


ON 


00 


b* 


b* 


Ul 


04 






• 


• 


• 


• 


• 


• 


• 


• 






On 


-tv 


Ul 


ON 


ON 


v4 


ON 


ON 


ON 




On 


00 


Ul 




VO 


-tv 


4V 


o 






• 


• 


• 


• 


• 


• 


• 


• 






-tv 


-fV 


4V 


Ul 


ON 


v4 


Ul 


On 


v4 




ON 


o 


Ul 


00 


VI 


ro 


vo 


ro 






• 


• 


• 


• 


• 


• 


• 


• 




.f 


04 


04 


-IV 


Ui 


ON 


v4 


Ul 


-tv 


00 




VO 


VO 


O 


ON 


-tV 


o 


to 


v4 






• 


• 


• 


• 


• 


• 


• 


• 






04 


04 


04 


-iv 


ON 


ui 


4V 


-tv 


VO 




-|V 


-tV 


ON 


u> 


b* 


00 


VO 


04 






• 


• 


• 


• 


• 


• 


• 


• 


b* 




OJ 


04 


04 


04 


Ui 


ui 


-IV 


-IV 


© 




ro 


04 


Ul 


ON 


00 


ui 




O 






• 


• 


• 


• 


• 


• 


• 


• 


h-» 




04 


ro 


04 


04 


ui 


ui 


04 


04 


b* 




o 


ON 


H 


|N4 


04 


to 


VO 


00 






















< 






b* 


H 


M 


b* 


b* 


b* 


h-» 


ft 




VO 


b* 


O 


-IV 


U4 


v4 


to 




n 




• 


• 


• 


* 


• 


• 


• 


• 


b* 




04 


ro 


Ul 


VO 


ON 


Ul 


Ul 


00 


ft 




o 


o 


00 


Ul 


b* 


h-C 


b* 


ON 


ft 




















O 




















-ft 






















• 




• 








• 






o 






ON 




-c- 










o 


04 


• 


• On 


• 


• V4 


• 


• 


• 


• 


ft 


-tV 


00 


00 « 


00 


00 


00 


VO VO 


00 


00 


CO 


• 


b* 


Ul 00 


ro 


00 u> 


ON 


o 


Ul 


VO 


b* 


ON 


b* 


04 ON 


VI 


00 N> 


ui 


-tv 4V 


-IV 


v4 


in 


Ui 


VO 


to 


o 


ON ** 


00 


00 ON 


ON 


00 


rr 






* 




X* 




rt 






ft 


* 








. 




* 






ft 




















o 
























• 










14 






. 



H 

O 

ft 

ft 



ft 

ft 

H 



O 

c 



<5 

ft 



*1 


ft 


n 






ft 


ft 


K* 






O 


1 


ft 






ft 




ft 






O 


ft 


f> 






*i 


•ft 


ft 






ft 


CO 


> 








5ft 


o 










o . 








OQ 


o 








ft* 


c 








rr 


ft 


- 


H 




CO 


ft 




ft 




1 


ft 




or 




O 


a- 




H 




ft 






ft 




b* 


f-h 








v- 


O 




H 




1 


ri 


















ft 


O* 








H* 


V* 








OQ 










ft* 










rr 


ft 








ft 


o 








CL 


ft 










o 








CO 


n 








c 


co 








cr 










rr 


o 








ft 


rh 








CO 










rr 










ft 


, 







